12 job splitters #45
12 job splitters #45
Conversation
|
@soxofaan I'm confused, there is a lint check that fails on black but the black check is disabled in the workflow file? I also tried fixing the files with black but doesn't help |
|
@soxofaan Do you think I should be using multiprocessing instead of multithreading? In case GFMAP is running on a VM with multiple CPUs that would really accelerate possibly heavy post-job actions |
That's indeed weird. What might be the case is Gihub runs the workflow from main (instead of your branch) because you are making a pull request targeting main. |
If I remember correctly, threading and multiprocessing have a common API, so I guess it's most future proof to stay within this common API, so that you can switch when one approach doesn't cut it |
|
@kvantricht also corresponds to #43 |
Conflicts: examples/extraction_pipelines/S2_extraction_example.ipynb
…stac items, updated extraction use case and linked AUXILIARY AssetDefinition inside stac submodule
18 generate stac
|
Problem remaining with the "to_scl_dilation_mask", except that everything should be ready for merge |
is that something that needs to be fixed before merging or do we create another issue for that? What exactly was the problem? |
There was a problem hiding this comment.
Huge work done, nice! I added some comments/questions. Don't entirely get the STAC yet and how much it's tuned now to WorldCereal. Also band names still need to be updated there. In the end is this just the example belonging to Sentinel-2 extraction for WorldCereal?
What I didn't check but what's important as you know is proper testing. I trust you've been writing the proper tests along the way.
|
Created issue for to_scl_dilation_mask and for the STAC AssetDefinition |
Closes #11 and #12
MultiBackendJobManager. The job manager also implements post-job actions that are computed on other threads to keep the status update smooth.TODO before merging: